1. The Field of the Invention
The present invention relates to methods and systems for displaying images, and more particularly, to methods and systems for efficiently rendering and animating characters using a hardware graphics unit when treating each pixel sub-component as an independent luminance intensity source.
2. Background and Related Art
Display devices are commonly used to render images to a human viewer. The effective rendering of images to a human viewer is fundamental to television and many types of computing technology. Accordingly, display devices are associated with televisions and many computing systems.
Images are rendered to a viewer using thousands of pixels distributed in a grid pattern on a display device. The color and/or intensity values of each of the pixels may be adjusted in order to form the desired image. In a typical display device, the color that a user perceives as emitting from a single pixel is actually represented by multiple displaced color components. For example, in a RGB display device, there is one light source that emits exclusively the color red. Another separate light source exclusively emits the color green. Another separate light source exclusively emit the color blue. These light sources are called herein the red, green, and blue color components of a pixel.
For any given pixel, these color components are spatially offset. However, the spatial offset is sufficiently small that a typical human user is unable to distinguish the individual color components of a pixel. Instead, the light from the color components blends together so that the pixel is perceived to have a single color. This single pixel color may be adjusted by adjusting the intensity of the red, green, and blue color components of the pixel such that the pixel may achieve a wide variety of perceived colors. White may be achieved by having maximum intensities in the red, green, and blue color components. Conversely, black may be achieved by having minimum intensities in the red, green, and blue color components.
Typical television displays and computer monitors rely on each pixel having multiple spatially displaced addressable components, whether those components be red, green, and blue color components, or otherwise. The Liquid Crystal Display (LCD) display is an example of a display device that utilizes multiple distinctly addressable elements, referred to herein as pixel sub-elements or pixel sub-components, to represent each pixel of an image being displayed. For example, FIG. 1 illustrates a conventional portable computer 100, which comprises a housing 101, a disk drive 102, a keyboard 103, and a display 104. The display 104 may be, for example, an LCD display.
Normally, each pixel on a color LCD display is represented by a single pixel element, which usually comprises three non-square pixel subcomponents such as a red pixel sub-component, a green pixel sub-component, and a blue pixel sub-component. Thus, a set of RGB pixel sub-components together makes up a single pixel element. Conventional LCD displays comprise a series of RGB pixel sub-components that are commonly arranged to form stripes along the display. The RGB stripes normally run the entire length of the display in one direction. The resulting RGB stripes are sometimes referred to as “RGB striping”. Common LCD monitors used for computer applications, which are wider than they are tall, tend to have RGB stripes running in the vertical direction.
FIG. 2A illustrates a known LCD screen 200 comprising a plurality of rows (R1-R12) and columns (C1-C16) that may be represented on the display 104. Each row/column intersection forms a square (or a rectangle that is almost the same in height as in width), which represents one pixel element. FIG. 2B illustrates the upper left hand portion of the known display 200 in greater detail.
Note in FIG. 2B how each pixel element (e.g., the [R2, C1] pixel element), comprises three distinct sub-components, a red sub-component 206, a green sub-component 207 and a blue sub-component 208. Each known pixel sub-component 206, 207, 208 is approximately one third the width of a pixel while being equal, in height, to the height of a pixel. As illustrated in FIG. 2A and FIG. 2B, one known arrangement of RGB pixel sub-components 206, 207, 208 form what appear to be vertical color stripes down the display 200. Accordingly, the arrangement of ⅓ width color sub-components 206, 207, 208, in the known manner illustrated in FIGS. 2A and 2B, is sometimes called “vertical striping”. While only 12 rows and 16 columns are shown in FIG. 2A for purposes of illustration, common column×row ratios include, e.g., 640×480, 800×600, and 1024×768.
In addition to vertical striping, LCDs are manufactured with pixel sub-components arranged in several additional patterns including, e.g., zig-zags and a delta pattern common in camcorder view finders, or in horizontal striping in which the RGB pixel sub-components each have one third of the entire pixel height, and have the same width as the pixel. The features of the present invention can be used with such pixel sub-component arrangements. However, since the RGB vertical striping configuration is more common, the embodiments of the present invention will be explained in the context of using RGB vertically striped displays.
Traditionally, each set of pixel sub-components for a pixel element is treated as a single pixel unit. Accordingly, in known systems luminous intensity values for all the pixel sub-components of a pixel element are generated from the same portion of an image. Consider for example, the image represented by the grid 220 illustrated in FIG. 2C. In FIG. 2C, each square represents an area of an image which is to be represented by a single pixel element including a red, green and blue pixel sub-component of the corresponding square of the grid 220.
In FIG. 2C, a shaded circle is used to represent a single image sample from which luminous intensity values are generated. Note how a single sample 222 of the image 220 is used in known systems to generate the luminous intensity values for each of the red, green, and blue pixel sub-components 232, 233, 234. Thus, in known systems, the RGB pixel sub-components are generally used as a group to generate a single colored pixel corresponding to a single sample of the image to be represented.
The light from each pixel sub-component group effectively adds together to create the effect of a single color whose hue, saturation, and intensity depends on the value of each of the three pixel sub-components. Say, for example, each pixel sub-component has a potential intensity of between 0 and 255. If all three pixel sub-components are given 255 intensity, the eye perceives the pixel as being white. However, if all three pixel sub-components are given a value of 0, the eye perceives a black pixel. By varying the respective intensities of each pixel sub-component, it is possible to generate millions of colors in between these two extremes.
Since, a single sample is mapped to a triple of pixel sub-components which are each ⅓ of a pixel in width, spatial displacement of the left and right pixel sub-components occurs since the centers of these elements is ⅓ from the center of the sample. Consider, for example, that an image to be represented was a red cube with green and blue components equal to zero. As a result of the displacement between the sample and green image sub-component, when displayed on an LCD display of the type illustrated in FIG. 2A, the apparent position of the cube on the display will be shifted one third of a pixel to the left of its actual position. Similarly, a blue cube would appear to be displaced one third of a pixel to the right. Thus, conventional imaging techniques used with LCD screens can result in undesirable image displacement errors.
Text characters represent one type of image which is particularly difficult to accurately display given typical flat panel display resolutions of 72 or 96 dots (pixels) per inch (dpi). Such display resolutions are far lower than the 600 dpi supported by most printers and the even higher resolutions found in most commercially printed text such as books and magazines. Accordingly, smaller visual objects such as text characters may appear coarse when the image resolution is limited to the pixel resolution.
Indeed, conventional wisdom was that the image resolution was necessarily limited to the pixel resolution. However, a technique for improving the resolution to the resolution of the pixel sub-component is described in a U.S. patent application Ser. No. U.S. Pat. No. 6,188,385 B1, issued Feb. 13, 2001, to William Hill et al., and entitled “Method and Apparatus for Displaying Images Such As Text” (hereinafter referred to as the “Hill et al. patent”), which is incorporated herein by reference in its entirety. A display technology that incorporates at least some of the technology described in the Hill et al. patent is often referred to as CLEARTYPE®, which term is a registered trademark of Microsoft Corporation.
The Hill et al. patent describes a technology that treats each pixel sub-component as a separate independent luminous intensity source. This contrasts with the conventional technique of treating the set of RGB pixel sub-components for a given pixel as being a single luminous intensity source.
In other words, the Hill et al. patent describes that each image sample is used to generate the luminance intensity value for a single pixel sub-component. This contrasts with the conventional technique of generating all of the pixel sub-component values for a given pixel using a single image sample. Thus, the technology described in the Hill et al. patent allows for a display device with RGB vertical striping to have an effective horizontal resolution that is up to three times greater than the horizontal pixel resolution.
FIG. 3 illustrates a general functional flow that may be implemented by the computer 100 in order to render and rasterize text images on the display 104 using the technology described in the Hill et al. patent. Suppose for purposes of discussion, that an application running on the computer 100 instructs the computer's operating system that the letter i having a given font and point size, is to be rendered and rasterized on the display 104. The left column of FIG. 3 labeled under the heading “Functional Flow” illustrates the general functions that are implemented to render a text character using this technology. The right column of FIG. 3 under the heading “Example” represents the state of the character i after the corresponding function to the left is implemented.
The process begins with a character description 301, which describes the form of a character. This may be accomplished by using vector graphics, lines, points and curves, from which a high-resolution digital representation of the character may be derived. A typical operating system will have a number of different character descriptions corresponding to each character of each font. Element 311 shows the visual representation of the character description for the letter i. In addition to the text information, the operating system also has access to background color and layout information for the images that are currently being displayed, and brush color and transparency information that are to be applied to the text character during rendering.
With this character and display information, operation proceeds to scaling 302 where non-square scaling is performed as a function of the direction and/or number of pixel sub-components included in each pixel element. In particular, the vertical direction of the character described in the character description is scaled so as to meet the height requirements for the point size specified by the application. However, the horizontal direction is scaled at a rate three times greater than in the vertical direction. This allows for subsequent image processing operations to take advantage of the higher horizontal degree of resolution that can be achieved by using individual pixel sub-components as independent luminous intensity sources in a vertically striped display.
In the simplest case, the scaling in the horizontal direction is at a relative rate that is related to the number of pixel sub-components in a given pixel. In the RGB vertical striping display, there are three pixel sub-components in any given pixel. Accordingly, in the simplest case, scaling in the horizontal direction occurs at a rate approximately three times the rate of scaling in the vertical direction. This scaling may occur by manipulating the character description as appropriate. Element 312 shows the state of the character represented by the scaled character description. Note that in the illustrated case where the height of the character remains the same, the letter i is stretched horizontally by a factor of approximately three during scaling.
After scaling 302, operation proceeds to hinting 303. The term “grid-fitting” is sometimes used to describe the hinting process. Hinting involves the alignment of a scaled character within a grid. It also involves the distorting of image outlines so that the image better conforms to the shape of the grid. The grid is determined as a function of the physical size of a display device's pixel elements. Unlike earlier techniques that failed to take into consideration pixel sub-component boundaries during hinting, hinting 303 treats pixel sub-component boundaries as boundaries along which characters can and should be aligned or boundaries to which the outline of a character should be adjusted.
The hinting process involves aligning the scaled representation of a character within the grid along or within pixel and pixel sub-component boundaries in a manner intended to optimize the accurate display of the character using the available pixel sub-components. In many cases, this involves aligning the left edge of a character stem with a left pixel or sub-pixel component boundary and aligning the bottom of the character's base along a pixel or pixel sub-component boundary.
Experimental results have shown that in the case of vertical striping, characters with stems aligned so that the character stem has a blue or green left edge generally tend to be more legible than characters with stems aligned to have a red left edge. Accordingly, during hinting of characters to be displayed on a screen with vertical striping, blue or green left edges for stems are favored over red left edges.
During hinting 303, the scaled image 312 is first placed over a grid pattern as represented by grid layout 313A. The grid pattern is shown for four columns of pixels labeled C1 through C4 from left to right, and six rows of pixels labeled R1 through R6 from top to bottom. Note that boundaries between pixel sub-components are represented by dashed lines except where there is also a boundary between pixels. The pixel boundaries are represented as solid lines. Note that each pixel sub-components has a heading R, G, or B representing whether the column represents the red, green, or blue color, respectively.
During hinting 303, the left edge of the scaled i character is aligned along the R/G pixel sub-component boundary so that the left edge of the stern of the hinted character 312′ has a green left edge to promote legibility. The shape of the character is also adjusted as well as the position of the character on the grid. Character spacing adjustments are also made.
Once the hinting 303 is complete, operation proceeds to scan conversion 304, which involves the conversion of the scaled geometry representing a character into a bitmap image. Conventional scan conversion operations treat pixels as individual units into which a corresponding portion of the scaled image can be mapped. However, in accordance with the Hill et al. patent, each pixel sub-component is treated as a separate luminous intensity component into which a separate portion of the scaled image can be mapped.
Referring to FIG. 3, the scan conversion operation results in the bitmap image 314. Note how each pixel sub-component of bitmap image columns C1-C4 is determined from a different segment of the corresponding columns of the scaled hinted image 313B. This contrasts with the conventional technique of having all three pixel sub-component values for a given pixel generated from a single portion of an image. Note also how the bitmap image 314, comprises a ⅔ pixel width stem with a left edge aligned along a red/green pixel boundary. Notice also that a dot that is ⅔ of a pixel in width is used. Conventional text imaging techniques that treated each pixel as a single luminous intensity component would have resulted in a less accurate image having a stem a full pixel wide and a dot a full pixel in size.
Once the bitmap representation of the text (i.e., bitmap image 314) is generated during scan conversion 304, it may be output to a display adapter or processed further to perform color processing operations and/or color adjustments to enhance image quality. While the human eye is much more sensitive to luminance edges as opposed to image color (chrominance) edges, treating the RGB pixel sub-components as independent luminous intensity elements for purposes of image rendering can result in undesired color fringing effects. If, for instance, you remove red from an RGB set, a color fringing effect of cyan, the additive of green and blue, is likely to result.
Thus, the bitmap image 314 may be supplied to color processing 305, where image processing is performed to determine how far away from the desired brush color the bitmap image has strayed. If portions of the bitmap image have strayed more than a pre-selected amount from the desired brush color, adjustments in the intensity values of pixel sub-components are applied until the image portions are brought within an acceptable range of an average between the brush and background colors.
The bitmap image 314 is then applied via a blending operation to the existing background image. In particular, for a given pixel, let the red, green, and blue color intensities be given by glyph.r, glyph.g, and glyph.b. A glyph is a term that represent the shape of the character with respect to that pixel sub-components of the given pixel. The three value vector of red, green, and blue color components is represented by the vector glyph.rgb.
The brush or foreground color components are represented by a similar vector brush.rgb. A scalar value of the transparency of the brush at each color component is given by the vector brusha.rgb. The background color for that pixel is given by a three value vector dst.rgb. In order to blend the brushed character onto the background, the following vector equation (1) is applied:
 DST.rgb=DST.rgb+(brush.rgb−dst.rgb)*glyph.rgb*brusha.rgb  (1)
In conventional techniques that treat each pixel sub-component as a separate and distinct luminance intensity value, this blending operation, as well as animations of the character (e.g., rotation and scaling) are performed in software. The calculations for performing the blending and animation of a character are quite complex. Even modem computing systems may be challenged by rendering and animating characters that treat each pixel sub-component as an independent luminance intensity source.
Accordingly, what is desired are systems and methods for rendering and animating characters that treat each pixel sub-component as an independent luminance intensity source in a more efficient manner.